Discrete Data Page Changes

• Fetch the CTD cast and summary sample files from the server directly rather than reading an index of separate files. The samples now include a type to indicate whether they came from the CTD cast file or the summary samples file.
• CTD cast data is stored under Assets > "CTD Cast (station-name)" in the "Asset" selector. Other strategies for handling this were very unintuitive, as having a split between instruments having an "Asset" name and CTD Casts only having a "Station" was very confusing. It seemed best to keep the filtering to just "Asset", though that may not be the best word anymore.
• Moved the R^2 value to the top right of the chart so it doesn't take up so much space.
• Changed the chart legend to "scroll" mode so it doesn't start overflowing the chart when there are a lot of series to defined.
• Added "Remove" and "Remove Other Assets" actions to the series' "..." menus.

Backend Changes

• Add scripts to /scripts directory to help with fetching and compiling data for the frontend to display.

  • ./scripts/refresh-discrete-data.sh: Fetch CTD cast data and discrete summary files from https://rawdata-west.oceanobservatories.org/files/ and subsequently parse them into files which can be read from the frontend: dashboard/public/discrete/ctd-cast-samples.csv and dashboard/public/discrete/summary-samples.csv respectively, which have both been added to the S3 bucket. This script uses the following scripts in its implementation:

  • ./scripts/scripts/download_rawdata.py: Downloading subsets of the file tree from https://rawdata-west.oceanobservatories.org/files/ into a local directory. Includes a --flatten flag which will include the paths of the files in the file names themselves, allowing a flat downloaded directory. Takes a glob as its first command line argument.

  • ./scripts/collect_discrete_ctd_cast_samples.py: Read .btl files from a directory and parse them into a single large CSV or (JSON file if specified) sample file which can be read from the frontend. This uses a lot of very tedious parsing code and a lot of hackery to determine which "Station" and "Target Asset" to associate samples with, considering the "Station"/"Location" are almost never specified in the same way in the bottle files. Claude Code was used to figure out most of the initial parsing and name mapping which would have taken forever otherwise.

  • ./scripts/collect_discrete_summary_samples.py: Read discrete data .csv files from a directory and combine them into a single large CSV or (JSON file if specified) sample file which can be read from the frontend. This one doesn't have much for fancy logic, it mostly just concatenates the files and works in a similar way to the CTD cast collection script.